Modern software applications and systems can be extremely complex, and may contain thousands or millions of lines of interrelated code spread across numerous files. Modern programs are also more interdependent than ever on other programs. That is because internet-based and other networked or otherwise connected systems continue to supplant stand-alone applications. Such networked applications may depend on many other programs to pass them appropriate data in order to run properly and without errors. Complex software systems carry with them a great risk of errors, such as so-called bugs.
Software generally is subjected to a number of iterative revisions as it moves from conception to initial launch (e.g., in alpha or beta test), and then through to commercial release. The process of identifying and tracking problems in software code as it goes through this process is generally referenced as quality assurance (QA), and entire departments may be devoted to such a function. One way that QA engineers attempt to identify problems in software for later eradication is by running a so-called “test case” by which software is run to see if its operates correctly and without errors.
A written test case can include a description of the functionality to be tested (taken, e.g., from the requirements for the software, or from “use cases,” such as specific examples of how the software is intended to be used), and the expected output of the case, so that the functionality of the software can be verified as working correctly. The engineer may then run the software to check its operation against the expected operation. As a simple example, an engineer may test a piece of software for adding two numbers together by running the software using two exemplary numbers, and checking whether the output matches the output that is expected. As a more realistic example, an engineer may place information into a database that is to be accessed by a program, and then may query the database using the software program to see if the appropriate data is returned and presented.
Each version of compiled and tested software is often referenced as a “build” of the software. Test cases generally need to be run for every build. A collection of related test cases is often referenced as a test suite.
Generally, QA engineers can use a test case management system during testing to help them plan testing, run the tests or test cases, and report the results of the testing. For the plan, the system may create trackable and loggable test cases to test one or more functionalities of an application, and to associate test cases with a particular bug or feature. For the run, the system may determine a sequence of test cases that need to be run, and may pass through the test cases and log the results of the testing in a database. For the reporting, the system may find information about test cases that were run on a specific “build” of the software, and may track progress of software during the QA cycle. The reporting may also report on code coverage, along with load and performance statistics and test progress, among other things.